Discovery of ingress provider edge devices in egress peering networks

ABSTRACT

The present disclosure generally discloses improvements in computer performance in communication networks, which may include improvements in computer performance in supporting discovery of ingress provider edge (PE) devices in a communication network using egress peer engineering (EPE) and software defined networking (SDN). The ingress PE devices of a network may have peering relationships with an egress peer link of an egress peer node of the network. The ingress PE devices may be discovered by identifying a set of traffic flows on an egress peer link of an egress peer node, identifying a set of ingress PE devices that have peering relationships with the egress peer node, determining bandwidth usage information, selecting, from the set of ingress PE devices based on the bandwidth usage information, a set of selected ingress PE devices, and initiating a management action for at least one of the selected ingress PE devices.

TECHNICAL FIELD

The present disclosure relates generally to communication networks and,more particularly but not exclusively, to providing various improvementsin computer performance for discovery of ingress provider edge devicesin a communication network using egress peer engineering and softwaredefined networking.

BACKGROUND

In communication networks, various combinations of communicationstechnologies may be employed in order to support communication of databetween endpoints. For example, egress peer engineering (EPE), in whichan ingress provider edge (PE) router may be instructed to use aparticular egress peer router and associated external interface in orderto reach a particular destination, may be used to support communicationof data between a source and the destination. Additionally, for example,software defined networking (SDN), in which the control plane isdisassociated from the data plane, may be used to support configurationof a communication network to use egress peer engineering (e.g., viainstallation of flow rules in ingress PE routers and egress peerrouters).

SUMMARY

The present disclosure generally discloses improvements in computerperformance in communication networks, including improvements incomputer performance for supporting the discovery of ingress provideredge devices in a communication network using egress peer engineering(EPE) and software defined networking (SDN).

In at least some embodiments, an apparatus for improving computerperformance is provided. The apparatus includes a processor and a memorycommunicatively connected to the processor. The processor is configuredto identify a traffic flow on an egress peer link of an egress peernode. The processor is configured to identify a set of ingress provideredge (PE) devices that have peering relationships with the egress peernode. The processor is configured to determine, for the ingress PEdevices based on respective sets of flow statistics associated with therespective ingress PE devices, respective bandwidth usages of the egresspeer link by the respective ingress PE devices for the traffic flow. Theprocessor is configured to select, from the set of ingress PE devicesbased on the respective bandwidth usages of the egress peer link by therespective ingress PE devices for the traffic flow, a selected ingressPE device. The processor is configured to initiate a management actionfor the selected ingress PE device. In at least some embodiments, acorresponding method is provided. In at least some embodiments, anon-transitory computer-readable storage medium stores instructionswhich, when executed by a computer, cause the computer to perform acorresponding method.

In at least some embodiments, an apparatus is provided. The apparatusincludes a processor and a memory communicatively connected to theprocessor. The processor is configured to identify a set of trafficflows on an egress peer link of an egress peer node. The processor isconfigured to identify a set of ingress provider edge (PE) devices thathave peering relationships with the egress peer node. The processor isconfigured to determine bandwidth usage information including, for eachcombination of each of the traffic flows and each of the ingress PEdevices, a respective bandwidth usage of the egress peer link by therespective ingress PE device for the respective traffic flow. Theprocessor is configured to select, from a set of ingress PE devicesbased on the bandwidth usage information, a set of selected ingress PEdevices. The processor is configured to initiate a management action forone of the selected ingress PE devices. In at least some embodiments, acorresponding method is provided. In at least some embodiments, anon-transitory computer-readable storage medium stores instructionswhich, when executed by a computer, cause the computer to perform acorresponding method.

In at least some embodiments, an apparatus is provided. The apparatusincludes a processor and a memory communicatively connected to theprocessor. The processor is configured to send, from an ingress provideredge (PE) device toward a first egress peer link of an egress peer nodehaving a peering relationship with the ingress PE device, a first datapacket of a traffic flow. The processor is configured to receive, from acontroller, a control message indicative of a redirection of the trafficflow from the first egress peer link to a second egress peer linkassociated with the first egress peer link. The processor is configuredto send, from the ingress PE router toward the second egress peer link,a second data packet of the traffic flow. In at least some embodiments,a corresponding method is provided. In at least some embodiments, anon-transitory computer-readable storage medium stores instructionswhich, when executed by a computer, cause the computer to perform acorresponding method.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings herein can be readily understood by considering thefollowing detailed description in conjunction with the accompanyingdrawings, in which:

FIG. 1 depicts an example communication system for illustratingdiscovery of a set of ingress provider edge devices providing traffic toan egress peer link of an egress peer node for a set of traffic flows;

FIG. 2 depicts the communication system of FIG. 1 for illustrating anexample of discovery of a set of ingress provider edge devices providingtraffic to an egress peer link for a set of traffic flows;

FIG. 3 depicts a method for use by an SDN controller in discovering aset of ingress provider edge devices providing traffic to an egress peerlink for a set of traffic flows;

FIG. 4 depicts a method for use by an SDN controller in discovering aset of ingress provider edge devices providing traffic to an egress peerlink for a traffic flow;

FIG. 5 depicts a method for use by an ingress provider edge device forredirecting traffic of a traffic flow from a first egress peer link to asecond egress peer link; and

FIG. 6 depicts a high-level block diagram of a computer suitable for usein performing various functions presented herein.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe figures.

DETAILED DESCRIPTION

The present disclosure generally discloses improvements in computerperformance in communication networks, including improvements incomputer performance to support the discovery of ingress provider edge(PE) devices in communication networks using egress peer engineering(EPE) and software defined networking (SDN). The discovery of ingress PEdevices may include discovery of a set of ingress PE devices providingdata to an egress peer link of an egress peer node for a set of trafficflows. The discovery of ingress PE devices may include, for a set oftraffic flows on an egress peer link (e.g., the top N traffic flows onthe egress peer link, where N≥1), discovery of a set of ingress PEdevices directing traffic of the traffic flows to the egress peer link(e.g., the top M ingress PE devices that are directing traffic of thetraffic flows to the egress peer link, where M≥1). The discovery of theset of ingress PE devices directing traffic of the traffic flows to theegress peer link may include identifying the set of traffic flows on theegress peer link (e.g., the top N traffic flows), identifying a set ofingress PE devices associated with the egress peering node (which mayinclude ingress PE devices directing traffic of the traffic flows to theegress peer link as well as ingress PE devices directing traffic ofother traffic flows to the egress peer link), and identifying the set ofingress PE devices directing traffic of the traffic flows to the egresspeer link (e.g., the top M ingress PE devices out of the set of ingressPE devices associated with the egress peer link). The discovery of theingress PE devices may be used for various management purpose (e.g.,intelligent traffic routing, intelligent traffic balancing, or thelike). It will be appreciated that these and various other embodimentsand potential advantages of ingress PE device discovery capabilities maybe further understood by way of reference to the example communicationnetwork of FIG. 1.

FIG. 1 depicts an example communication system for illustratingdiscovery of a set of ingress provider edge devices providing traffic toan egress peer link of an egress peer node for a set of traffic flows.

The communication system 100 includes a network 110, a network 120, anetwork 130, and a software defined networking (SDN) controller 140. TheSDN controller 140 is configured to provide control functions for thenetwork 110.

The network 110, the network 120, and the network 130 may be based onvarious communications technologies. For example, the network 110 may bean Internet Protocol (IP)/Multiprotocol Label Switching (MPLS) network.For example, the network 120 may be an IP network, an Ethernet network,or the like, as well as various combinations thereof. For example, thenetwork 130 may be an IP network, an Ethernet network, or the like, aswell as various combinations thereof. The network 110, the network 120,and the network 130 each may support various data plane technologies andcapabilities (e.g., elements, protocols, or the like, as well as variouscombinations thereof), control plane technologies and capabilities(e.g., elements, protocols, or the like, as well as various combinationsthereof), or the like, as well as various combinations thereof. Thenetwork 110, the network 120, and the network 130 may be based onvarious other communications technologies.

The network 110, the network 120, and the network 130 are configured asdifferent autonomous systems and, thus, have different autonomous systemnumbers assigned thereto. For example, the network 110 has an autonomoussystem number of 65000 assigned thereto, the network 120 has anautonomous system number of 65001 assigned thereto, and the network 130has an autonomous system number of 65002 assigned thereto. It will beappreciated that other autonomous system numbers may be used. It will beappreciated that the network 110, the network 120, and the network 130may be owned or operated by different entities.

The network 110, the network 120, and the network 130 are configured tosupport transport of traffic flows. The traffic flows may includevarious traffic flow types which may transport various types of traffic.For example, the traffic flows may include video flows that aretransporting video traffic, voice flows that are transporting voicetraffic, data flows that are transporting data, or the like, as well asvarious combinations thereof.

The network 110 includes a set of ingress PE devices 111-1-111-3(collectively, ingress PE devices 111). The ingress PE devices 111 maybe routers, switches, or other suitable types of PE devices. The ingressPE devices 111 may be controlled by SDN controller 140. The ingress PEdevices 111 may include flow rules tables and the SDN controller 140 mayinstall flow rules into the flow rules table of the ingress PE devices111 (e.g., flow forwarding rules for use by ingress PE devices 111 inforwarding packets of traffic flows, flow statistics collection rulesfor controlling collection of flow statistics at the ingress PE devices111, or the like, as well as various combinations thereof). It is notedthat the ingress PE devices 111 are referred to as ingress devices asthe description primarily considers traffic flows proceeding in adirection from the network 110 toward the networks 120 and 130 (suchthat the ingress PE devices 111 are ingress points into the network 110for the traffic flows); however, it will be appreciated that traffic mayflow in both directions (and, thus, that, for traffic flows proceedingin a direction to the network 110 from the networks 120 and 130, theingress PE devices 111 may be referred to as egress PE devices or egresspeer nodes). It will be appreciated that traffic flows of network 110may be defined in various ways (which may depend on the underlyingcommunications technology of network 110), such as based onsource/destination address pairs (e.g., source IP address anddestination IP address), five-tuples (e.g., source IP address, sourceport number, destination IP address, destination port number, andprotocol), route prefixes, subnets, or the like.

The network 110 includes a set of egress border routers (BRs)119-1-119-2 (collectively, egress BRs 119). The egress BR 119-1 is aborder router that is connected to the network 120 via an egress peerlink 122. The egress BR 119-2 is a border router that is connected tothe network 130 via an egress peer link 132. The egress BRs 119 may beconsidered to be autonomous system BRs (ASBRs) as these routers areoperating at the border between networks operating as separateautonomous systems. The egress BRs 119 may be controlled by SDNcontroller 140. The egress BRs 119 may include flow rules tables and theSDN controller 140 may install flow rules into the flow rules table ofthe egress BRs 119 (e.g., flow forwarding rules for use by egress BRs119 in forwarding packets of traffic flows, flow statistics collectionrules for controlling collection of flow statistics at the egress BRs119, or the like, as well as various combinations thereof). It is notedthat the egress BRs 119 are referred to as egress devices as thedescription primarily considers traffic flows proceeding in a directionfrom the network 110 toward the networks 120 and 130 (such that the BRs119 are egress points from network 110 for the traffic flows); however,it will be appreciated that traffic may flow in both directions (and,thus, that, for traffic flows proceeding in a direction to the network110 from the networks 120 and 130, the egress BRs 119 may be referred toas ingress BRs or ingress PE devices).

The network 110 is configured to support egress peer engineering (EPE).In general, EPE is a network use case in which an ingress router (e.g.,an ingress PE device 111) is instructed to use a specific egress peerrouter (e.g., egress BR 119-1 or egress BR 119-2) and a specificexternal interface (e.g., egress peer link 122 for egress BR 119-1 oregress peer link 132 for egress BR 119-2) to reach a particulardestination. The EPE capabilities of network 110 may be based on arouting protocol, such as the Border Gateway Protocol (BGP) or othersuitable type of protocol. The EPE capabilities of network 110 may bebased on Segment Routing (SR) route advertisement. For example, the EPEcapabilities of network 110 may be provided using one or more of SegmentRouting Egress Peer Engineering BGP-LS Extensions (BGP-LS-EPE], SegmentRouting Centralized Egress Peer Engineering [SR-EPE], or the like, aswell as various combinations thereof. The network 110 is configured,based on EPE, such that each of the ingress PE devices 111 is associatedwith one of the egress BRs 119 such that the one of the egress BRs 119with which the respective ingress PE device 111 is associated is anegress peer node for the respective ingress PE device 111. In thenetwork 110, the egress BR 119-1 is the primary egress peer node foreach of the ingress PE devices 111-1, 111-2, and 111-3, and, thus, theingress PE devices 111-1, 111-2, and 111-3 may be considered to havepeering relationships with egress BR 119-1. Similarly, in the network110, the egress peer link 122 is the primary egress peer link for eachof the ingress PE devices 111-1, 111-2, and 111-3, and, thus, theingress PE devices 111-1, 111-2, and 111-3 may be considered to havepeering relationships with egress peer link 122. In the network 110, theegress BR 119-2 is available as an alternate egress peer node for eachof the ingress PE devices 111-1, 111-2, and 111-3. The egress BR 119-2may be an alternate egress peer node for the ingress PE devices 111-1,111-2, and 111-3 in the sense that the destinations of the traffic flowsfrom the ingress PE devices 111-1, 111-2, and 111-3 are reachable viaboth egress BR 119-1 and egress BR 119-2. Similarly, in the network 110,the egress peer link 132 is available as an alternate egress peer linkfor each of the ingress PE devices 111-1, 111-2, and 111-3. It will beappreciated that, where EPE is based on BGP, the egress BRs 119 may beconsidered to be egress BGP peers nodes.

The network 110, as indicated above, is configured to support SDN. Theingress PE devices 111 and the egress BRs 119 may be configured tosupport communication with SDN controller 140. The ingress PE devices111 and the egress BRs 119 may be controlled by the SDN controller 140using SDN capabilities. For example, as indicated above, the ingress PEdevices 111 and the egress BRs 119 may be configured with flow rulestables for providing various functions within network 110 based on SDN.The SDN capabilities of the network 110 may be provided using Open Flowor other suitable mechanisms for supporting software defined networking.

The network 110 may be configured to support various other functions andfeatures. For example, network 110 may be configured to support BGP,which also may include support for BGP FlowSpec and other BGP-relatedcapabilities. For example, network 110 may be configured to supportInternet Protocol Flow Information Export (IPFIX). For example, network110 may be configured to support the Resource Reservation Protocol(RSVP), which also may include support for the RSVP-Traffic Engineering(RSVP-TE) tunnel technology and other RSVP-related capabilities. Thenetwork 110 may be configured to support various other functions andfeatures.

The network 110 may include various other network elements andcommunication links, which have been omitted from FIG. 1 for purposes ofclarity.

The network 120 is a communication network configured to supportcommunications of network 110. The network 120 includes a providerrouter 121. The provider router 121 of network 120 is connected to theegress BR 119-1 of network 110 via an egress peer link 122. The providerrouter 121 may be an ingress PE device configured to operate as aningress point into network 120, from the egress BR 119-1 of the network110, for traffic flows flowing in a direction from the network 110 tothe network 120. The network 120 may be configured to transport thetraffic flows received from the egress BR 119-1 for delivery to theintended destination(s) of the traffic flows. It will be appreciatedthat, although omitted from FIG. 1 for purposes clarity, network 120 mayinclude various other network elements (e.g., routers, switches,controllers, or the like), communication links, or the like, as well asvarious combinations thereof.

The network 130 is a communication network configured to supportcommunications of network 110. The network 130 includes a providerrouter 131. The provider router 131 of network 130 is connected to theegress BR 119-2 of network 110 via an egress peer link 132. The providerrouter 131 may be an ingress PE device configured to operate as aningress point into network 130, from the egress BR 119-2 of the network110, for traffic flows flowing in a direction from the network 110 tothe network 130. The network 130 may be configured to transport thetraffic flows received from the egress BR 119-2 for delivery to theintended destination(s) of the traffic flows. It will be appreciatedthat, although omitted from FIG. 1 for purposes clarity, network 130 mayinclude various other network elements (e.g., routers, switches,controllers, or the like), communication links, or the like, as well asvarious combinations thereof.

The SDN controller 140 is configured to control the network 110 based onSDN, which may include controlling various elements of the network 110(illustratively, ingress PE devices 111 and egress BRs 119) based onSDN. The SDN controller 140 includes an ingress PE device discoveryelement 141. The ingress PE device discovery element 141 is configuredto perform various functions of the ingress PE device discoverycapability. The ingress PE device discovery element 141 may beconfigured to perform various functions for discovering a set of ingressPE devices 111 providing traffic to an egress BR 119 for a set oftraffic flows. The operation of SDN controller 140 and, morespecifically, the ingress PE discovery element 141, may be furtherunderstood by way of reference to the example of FIG. 2 and the methodsof FIGS. 3-4.

It will be appreciated that, although embodiments of the ingress PEdevice discovery capability are primarily presented herein within thecontext of a communication system using of specific numbers, types, andarrangements of networks and elements of those networks, variousembodiments of the ingress PE device discovery capability may beprovided within the context of communication systems using othernumbers, types, and/or arrangements of networks (or elements of thosenetworks).

FIG. 2 depicts the communication system of FIG. 1 for illustrating anexample of discovery of a set of ingress provider edge devices providingtraffic to an egress peer link for a set of traffic flows.

The communication system 100, as discussed above, includes the network110, the network 120, the network 130, and the SDN controller 140. TheSDN controller 140 is configured to discover a set of ingress PE devices(illustratively, a set of ingress PE devices 111) providing traffic toan egress peer link of an egress peer node (illustratively, egress peerlink 122 associated with egress BR 119-1 or egress peer link 132associated with egress BR 119-2) for a set of traffic flows.

The SDN controller 140 selects an egress peer link for which ingress PEdevice discovery is to be performed (which also may be referred to as anegress peer link of interest). The SDN controller 140 may select theegress peer link randomly, based on a schedule (e.g., using a periodic,round-robin cycling between selection of the various egress peer linkavailable for selection, responsive to an indication of a condition onor associated with the egress peer link (e.g., a congestion conditionindicative that the egress peer link is overloaded or is approaching anoverload condition), or the like, as well as various combinationsthereof. In the example of FIG. 2, it is assumed that the egress peerlink 122 of the egress BR 119-1 is selected as the egress peer link forwhich ingress PE device discovery is to be performed.

The SDN controller 140 identifies a set of traffic flows on the egresspeer node and the egress peer link for which ingress PE device discoveryis to be performed (which also may be referred to as a set of trafficflows of interest or, more simply, traffic flows of interest). Thetraffic flows on the egress peer link may include all of the trafficflows on the egress peer link or a subset of the traffic flows on theegress peer link. The traffic flows may be the top N (N≥1) traffic flowson the egress peer link, which may be measured in terms of bandwidth.The SDN controller 140 may identify the set of traffic flows on theegress peer link based on flow information associated with the egresspeer link. The flow information associated with the egress peer link mayinclude per-flow bandwidth information that includes, for each of thetraffic flows, an indication of the amount of bandwidth of therespective traffic flow that is supported by the egress peer link. Theper-flow bandwidth information may be based on flow statistics which maybe collected for the traffic flows on the egress peer node and theegress peer link. The flow information associated with the egress peerlink may be received by the SDN controller 140 from the egress peer node(e.g., where the egress peer node is configured to collect and processflow statistics for traffic flows traversing the egress peer node inorder to determine the flow information), determined by the SDNcontroller 140 based on information received by the SDN controller 140from the egress peer node (e.g., based on processing of flow statisticsreceived from the egress peer node), or the like, as well as variouscombinations thereof. In at least some embodiments, in which the egresspeer node provides the flow information to the SDN controller 140, theconfiguration of the egress peer node by the SDN controller 140 tocollect flow statistics and provide the flow information and theproviding of the flow information to the SDN controller 140 may be basedon a control protocol (e.g., OpenFlow, BGP FlowSpec, Netflow or thelike). In at least some embodiments, in which the egress peer nodeprovides flow statistics to the SDN controller 140 for use by the SDNcontroller to determine the flow information, the configuration of theegress peer node by the SDN controller 140 to collect flow statisticsand provide the flow statistics to the SDN controller 140 may be basedon a flow statistics collection protocol (e.g., Netflow IPFIX or thelike), a control protocol (e.g., OpenFlow, BGP FlowSpec, or the like),or the like, as well as various combinations thereof. The egress peernode may already be configured to provide information to the SDNcontroller 140 at the time that the egress peer node is selected by theSDN controller 140 (e.g., based on previous installation of rulesrelated to collection of flow statistics on the egress peer node), maybe configured by the SDN controller 140 to provide information to theSDN controller 140 based on selection of the egress peer node by the SDNcontroller 140 (e.g., the SDN controller 140 installs rules related tocollection of flow statistics on the egress peer node after selection ofthe egress peer node by the SDN controller 140), or the like. Theidentification of the set of traffic flows on the egress peer link mayinclude selection of the set of traffic flows on the egress peer linkfrom a set of available traffic flows on the egress peer link (e.g.,selection of a particular subset of traffic flows from the full set offlows on the egress peer link, such as selection of the top N trafficflows on the egress peer link in terms of bandwidth consumption). In theexample of FIG. 2, as indicated by the flow information of flowinformation table 201, the SDN controller 140 identifies the top threetraffic flows, in terms of bandwidth consumption, on egress peer link122 of egress BR 119-1. The top three traffic flows include a firsttraffic flow (which is indicated by a source/destination IP address pairof 1.1.1.0/24) for which egress peer link 122 of egress BR 119-1supports 2.5 M, a second traffic flow (which is indicated by asource/destination IP address pair of 2.2.1.0/24) for which egress peerlink 122 of BR 119-1 supports 2 M, and a third traffic flow (which isindicated by a source/destination IP address pair of 3.3.1.0/24) forwhich egress peer link 122 of BR 119-1 supports 1 M.

The SDN controller 140 identifies a set of ingress PE devices associatedwith the egress peer node. The set of ingress PE devices associated withthe egress peer node may include the ingress PE devices having peeringrelationships (e.g., based on EPE) with the egress peer node. The SDNcontroller 140 may identify the set of ingress PE devices associatedwith the egress peer node based on one or more routing protocols (e.g.,Border Gateway Protocol (BGP) or the like) running on the egress peernode. For example, where the egress peering node is running a routingprotocol, the routing protocol neighbors of the egress peering node(which include the ingress PE devices associated with the egress peernode) may be determined based on the routing protocol. The SDNcontroller 140 may identify the set of ingress PE devices associatedwith the egress peer node based on prefix information. In the example ofFIG. 2, the SDN controller 140 determines that ingress PE devices 111-1,111-2, and 111-3 are all associated with the egress BR 119-1 (and, thus,with the egress peer link 122 of the egress BR 119-1).

The SDN controller 140 determines, for each of the ingress PE devices, arespective set of bandwidth usages of the egress peer link by therespective ingress PE device for the respective traffic flows. Forexample, for a given ingress PE device, the respective set of bandwidthusages of the egress peer link by the given ingress PE device mayinclude, for each of the traffic flows for which the given ingress PEdevice is sourcing traffic to the egress peer link, an indication of anamount of bandwidth of the respective traffic flow that is propagatedfrom the given ingress PE device to the egress peer link (whichindicates the contribution of the given ingress PE device to congestionon the egress peer link due to the respective traffic flow). The sets ofbandwidth usages of the egress peer link by the respective ingress PEdevices may be determined based on respective sets of flow statistics ofthe respective ingress PE devices. For example, for a given ingress PEdevice, the respective set of bandwidth usages of the egress peer linkby the given ingress PE device may be determined based on a respectiveset of flow statistics of the respective ingress PE device. Therespective set of flow statistics of the respective ingress PE devicerelated to the traffic flows provided from the respective ingress PEdevice to the egress peer link may be receive by the SDN controller fromthe given ingress PE device (e.g., based on flow statistics collectionrules previously installed on the ingress PE device by the SDNcontroller 140, based on flow statistics collection rules installed onthe ingress PE device responsive to identification of the given ingressPE device as sourcing traffic to the egress peer link for one or more ofthe traffic flows, or the like, as well as various combinationsthereof). The installation of the flow statistics collection rules andassociated reporting of the flow statistics by the ingress PE devicesmay be performed based on a flow-based protocol (e.g., OpenFlow, BGPFlowSpec, or the like). In the example of FIG. 2, as indicated by theflow information of flow information table 202-1 associated with ingressPE device 111-1, the SDN controller 140 determines, for each of thetraffic flows traversing the egress peer link 122 of the egress BR 119-1that is sourced from the ingress PE device 111-1, an indication of anamount of bandwidth of that traffic flow that is sent from ingress PEdevice 111-1 to the egress peer link 122 of the egress BR 119-1 (e.g.,for the traffic flow denoted using 1.1.1.0/24 the ingress PE device111-1 sends 1 M of data to the egress peer link 122 of the egress BR119-1 and for the traffic flow denoted using 2.2.1.0/24 the ingress PEdevice 111-1 sends 1.5 M of data to the egress peer link 122 of theegress BR 119-1). Similarly, in the example of FIG. 2, as indicated bythe flow information of flow information table 202-2 associated withingress PE device 111-2, the SDN controller 140 determines, for each ofthe traffic flows traversing the egress peer link 122 of the egress BR119-1 that is sourced from the ingress PE device 111-2, an indication ofan amount of bandwidth of that traffic flow that is sent from theingress PE device 111-2 to the egress peer link 122 of the egress BR119-1 (e.g., for the traffic flow denoted using 1.1.1.0/24 the ingressPE device 111-2 sends 1 M of data to the egress peer link 122 of theegress BR 119-1 and for the traffic flow denoted using 2.2.1.0/24 theingress PE device 111-2 sends 0.5 M of data to the egress peer link 122of the egress BR 119-1).

The SDN controller 140 determines, based on the respective sets ofbandwidth usages of the egress peer link by the respective ingress PEdevices, per-PE/per-flow bandwidth usage information for the egress peerlink (which, for at least some embodiments, also may be referred to moregenerally herein as bandwidth usage information). The per-PE/per-flowbandwidth usage information for the egress peer link includes, for eachcombination of each traffic flow and each ingress PE device providingtraffic of the traffic flow to the egress peer link, a respectiveindication of the amount of bandwidth provided from the respectiveingress PE device to the egress peer link for the respective trafficflow. In the example of FIG. 2, as indicated by the flow information offlow information table 203 (determined based on the flow information offlow information tables 202), the per-PE/per-flow bandwidth usageinformation includes an indication that ingress PE device 111-2 issending 1.5 M of data to the egress peer link 122 of the egress BR 119-1for traffic flow 2.2.1.0/24, ingress PE device 111-1 is sending 1 M ofdata to the egress peer link 122 of the egress BR 119-1 for traffic flow1.1.1.0/24, ingress PE device 111-2 is sending 1 M of data to the egresspeer link 122 of the egress BR 119-1 for traffic flow 1.1.1.0/24, andingress PE device 111-2 is sending 0.5 M of data to the egress peer link122 of the egress BR 119-1 for traffic flow 2.2.1.0/24.

The SDN controller 140 may initiate various management actions based onbandwidth usage information (e.g., the per-PE/per-flow bandwidth usageinformation for the egress peer link). It is noted that examples of somemanagement actions which may be provided are discussed further below.

The SDN controller 140, for a given traffic flow contributing tocongestion on the egress peer link (e.g., a traffic flow on the egresspeer link that is identified as causing congestion on the egress peerlink), may identify a set of ingress PE devices that are contributing tocongestion of the given traffic flow on the egress peer link. The set ofingress PE devices that are contributing to congestion of the giventraffic flow on the egress peer link may include all of the ingress PEdevices sending traffic of the traffic flow to the egress peer link(which may include all or some of the ingress PE devices identified asbeing associated with the egress peer link) or a subset of the ingressPE devices sending traffic of the traffic flow to the egress peer link.The ingress PE devices sending traffic of the traffic flow to the egresspeer link may include the top X (X≥1) ingress PE devices sending trafficof the traffic flow to the egress peer link. The SDN controller 140 mayinitiate a management action to attempt to alleviate the congestion onthe egress peer link due to the traffic flow. The SDN controller 140 mayselect, from the set of ingress PE devices that are contributing tocongestion of the given traffic flow on the egress peer link, a selectedingress PE device sending traffic to the egress peer link for the giventraffic flow (e.g., the ingress PE device sending the most traffic tothe egress peer link for the given traffic flow, the ingress PE devicesending the next-to-most traffic to the egress peer link for the giventraffic flow, or the like) and may initiate redirection of the trafficof the traffic flow from being sent from the selected ingress PE deviceto the egress peer link to being sent from the selected ingress PEdevice to an alternate egress peer link (e.g., of the same egress peernode or a different egress peer node). The SDN controller 140 mayinitiate redirection of the traffic of the traffic flow from being sentfrom the selected ingress PE device to the egress peer link to beingsent from the selected ingress PE device to an alternate egress peerlink by installing one or more flow forwarding rules for that trafficflow on the selected ingress PE device such that the selected ingress PEdevice forwards packets of the traffic flow to the alternate egress peerlink rather than the egress peer link. The SDN controller 140 mayinitiate redirection of the traffic of the traffic flow from being sentfrom the selected ingress PE device to the egress peer link to beingsent from the selected ingress PE device to an alternate egress peerlink based on segment routing (e.g., using a segment routing tunnel),based on RSVP-TE (e.g., using an RSVP-TE tunnel), or the like, as wellas various combinations thereof. In the example of FIG. 2, the SDNcontroller 140 may initiate redirection of a portion of the traffic ofthe traffic flow 2.2.1.0/24 from the egress peer link 122 of the egressBR 119-1 to the egress peer link 132 of the egress BR 119-2 which is thealternate egress peer link for the egress peer link 122 of the egress BR119-1. The SDN controller 140, where it is determined that the trafficflow 2.2.1.0/24 is contributing to congestion on the egress peer link122 of the egress BR 119-1, may use the per-PE/per-flow bandwidth usageinformation in flow information table 203 to determine that ingress PEdevice 111-1 and ingress PE device 111-2 are both sourcing traffic ofthe traffic flow 2.2.1.0/24 to the egress peer link 122 of the egress BR119-1 and that ingress PE device 111-1 is sourcing more traffic oftraffic flow 2.2.1.0/24 to the egress peer link 122 of the egress BR119-1 (namely, 1.5 M) than the ingress PE device 111-2 (namely, 0.5 M).The SDN controller 140 may then configure ingress PE device 111-1 toredirect the traffic of the traffic flow 2.2.1.0/24 from being sent fromingress PE device 111-1 to the egress peer link 122 of the egress BR119-1 to being sent from ingress PE device 111-1 to the egress peer link132 of the egress BR 119-2. It is noted that, although primarilypresented with respect to embodiments in which only a single ingress PEdevice sending traffic to the egress peer link for the given trafficflow is selected, in at least some embodiments multiple ingress PEdevices sending traffic to the egress peer link may be selected forredirection of traffic of the traffic flow (e.g., the two ingress PEdevices sending the greatest amount traffic to the egress peer link forthe traffic flow, the three egress peer nodes sending the lowest amountof traffic to the egress peer link for the traffic flow, or the like).It will be appreciated that other selections for application ofmanagement actions are contemplated.

The SDN controller 140, for an egress peer link experiencing congestion,may identify a set of ingress PE devices that are contributing tocongestion on the egress peer link. The set of ingress PE devices thatare contributing to congestion on the egress peer link may include allof the ingress PE devices sending traffic to the egress peer link (whichmay include all or some of the ingress PE devices identified as beingassociated with the egress peer link) or a subset of the ingress PEdevices sending traffic to the egress peer link (e.g., ones of theingress PE devices sending an amount of traffic to the egress peer linkthat is greater than a threshold). The ingress PE devices sendingtraffic to the egress peer link may include the top X (X≥1) ingress PEdevices sending traffic to the egress peer link. The SDN controller 140may initiate a management action to attempt to alleviate the congestionon the egress peer link. The SDN controller 140 may select, from the setof ingress PE devices that are contributing to congestion on the egresspeer link, a selected ingress PE device sending traffic to the egresspeer link (e.g., the ingress PE device sending the most traffic to theegress peer link, the ingress PE device sending the next-to-most trafficto the egress peer link, or the like) and may initiate redirection oftraffic from being sent from the selected ingress PE device to theegress peer link to being sent from the selected ingress PE device to analternate egress peer link (e.g., of the same egress peer node or adifferent egress peer node). The SDN controller 140 may initiateredirection of traffic from being sent from the selected ingress PEdevice to the egress peer link to being sent from the selected ingressPE device to an alternate egress peer link by installing one or moreflow forwarding rules on the selected ingress PE device for one or moretraffic flows (e.g., for all traffic flows being sent from the selectedingress PE device to the egress peerlink, for a subset of traffic flowsbeing sent from the selected ingress PE device to the egress peer link,or the like) such that the selected ingress PE device forwards packetsof the one or more traffic flows to the alternate egress peer linkrather than the egress peer link. In the example of FIG. 2, the SDNcontroller 140 may initiate redirection of the traffic of traffic flows1.1.1.0/24 and 2.2.1.0/24 from being sent from ingress PE device 111-1to the egress peer link 122 of the egress BR 119-1 to being sent fromingress PE device 111-1 to the egress peer link 132 of the egress BR119-2. The SDN controller 140, where it is determined that the egresspeer link 122 of the egress BR 119-1 is experiencing congestion, may usethe per-PE/per-flow bandwidth usage information in flow informationtable 203 to determine that ingress PE device 111-1 and ingress PEdevice 111-2 are both sourcing traffic to the egress peer link 122 ofthe egress BR 119-1 and that ingress PE device 111-1 is sourcing moretraffic to the egress peer link 122 of the egress BR 119-1 (namely, 2.5M) than ingress PE device 111-2 (namely, 1.5 M). The SDN controller 140may then configure ingress PE device 111-1 to redirect the traffic oftraffic flows 1.1.1.0/24 and 2.2.1.0/24 from being sent from ingress PEdevice 111-1 to the egress peer link 122 of the egress BR 119-1 to beingsent from ingress PE device 111-1 to the egress peer link 132 of theegress BR 119-2. It is noted that, although primarily presented withrespect to embodiments in which only a single ingress PE device sendingtraffic to the egress peer link for the given traffic flow is selected,in at least some embodiments multiple ingress PE devices sending trafficto the egress peer link may be selected for redirection of traffic(e.g., the two ingress PE devices sending the greatest amount traffic tothe egress peer link for the traffic flow, the three egress peer nodessending the lowest amount of traffic to the egress peer link for thetraffic flow, or the like). It will be appreciated that other selectionsfor application of management actions are contemplated.

The SDN controller 140 may be configured to provide various other typesof management actions based on the per-PE/per-flow bandwidth usageinformation for the egress peer link.

FIG. 3 depicts a method for use by an SDN controller in discovering aset of ingress provider edge devices providing traffic to an egress peernode for a set of traffic flows. It is noted that, although primarilypresented in FIG. 3 as being performed serially, at least a portion ofthe functions of method 300 of FIG. 3 may be performed contemporaneouslyor in a different order than as presented in FIG. 3.

At block 301, method 300 begins.

At block 310, the SDN controller selects an egress peer link, of anegress peer node, for which ingress PE device discovery is to beperformed.

At block 320, the SDN controller identifies a set of traffic flows onthe egress peer link for which ingress PE device discovery is to beperformed. This may include all of the traffic flows on the egress peerlink or a subset of traffic flows on the egress peer link.

At block 330, the SDN controller identifies a set of ingress PE devicesassociated with the egress peer node. This may include a set of ingressPE devices peering with the egress peer node based on egress peerengineering. This may include a set of ingress PE devices peering withthe egress peer link of the egress peer node based on egress peerengineering.

At block 340, the SDN controller determines, for each of the ingress PEdevices, a respective set of bandwidth usages of the egress peer link bythe respective ingress PE device for the respective traffic flows.

At block 350, the SDN controller determines, based on the respectivesets of bandwidth usages of the egress peer link by the respectiveingress PE devices, per-PE/per-flow bandwidth usage information for theegress peer link.

At block 360, the SDN controller selects, based on the per-PE/per-flowbandwidth usage information for the egress peer link, a selected ingressPE device. The selected ingress PE device is one of the ingress PEdevices for which a management action is to be performed. The selectedingress PE device may be one of the ingress PE devices sourcing the mosttraffic to the egress peer link for one of the traffic flows (e.g., oneof the traffic flows causing the most congestion on the egress peerlink), one of the ingress PE devices sourcing the most traffic to theegress peer link, or the like.

At block 370, the SDN controller initiates a management action for theselected ingress PE device. The management action that is initiated forthe selected ingress PE device may depend on the basis for selection ofthe selected ingress PE device. The management action may includeredirecting traffic of a particular traffic flow (e.g., the traffic flowcausing the most congestion on the egress peer link) from being providedfrom the selected ingress PE device to the egress peer link to beingprovided from the selected ingress PE devices to an alternate egresspeer link associated with the egress peer link, redirecting traffic ofmultiple traffic flows (e.g., multiple traffic flows contributing tocongestion on the egress peer link) from being provided from theselected ingress PE device to the egress peer link to being providedfrom the selected ingress PE devices to an alternate egress peer link,or the like. It will be appreciated that various other managementactions may be initiated by the SDN controller for the selected ingressPE device.

At block 399, method 300 ends.

It will be appreciated that, although primarily directed to embodimentsin which a single ingress PE device is selected for initiation of amanagement action, method 300 of FIG. 3 may be adapted such thatmultiple ingress PE devices may be selected for initiation of multiplemanagement actions (e.g., redirecting traffic of a traffic flow frommultiple ingress PE devices to one or more alternate egress peer link,redirecting traffic of multiple traffic flows from multiple ingress PEdevices to one or more alternate egress peer link, or the like).

It will be appreciated that, although method 300 of FIG. 3 is primarilydirected to discovery of a set of ingress PE devices providing trafficto an egress peer link for a set of traffic flows, the set of trafficflows may include a single traffic flow on the egress peer link (e.g., atraffic flow of interest, such as a traffic flow identified as causingor contributing to congestion on the egress peer node). Accordingly, anembodiment of ingress PE device discovery of a set of ingress PE devicesproviding traffic to an egress peer link for a single traffic flow ispresented with respect to FIG. 4.

FIG. 4 depicts a method for use by an SDN controller in discovering aset of ingress provider edge devices providing traffic to an egress peernode for a traffic flow. It is noted that, although primarily presentedin FIG. 4 as being performed serially, at least a portion of thefunctions of method 400 of FIG. 4 may be performed contemporaneously orin a different order than as presented in FIG. 4.

At block 401, method 400 begins.

At block 410, the SDN controller identifies a traffic flow on an egresspeer link of an egress peer node.

At block 420, the SDN controller identifies a set of ingress PE deviceshaving a peering relationship with the egress peer node.

At block 430, the SDN controller determines, for the ingress PE devicesbased on respective sets of flow statistics associated with therespective ingress PE devices, respective bandwidth usages of the egresspeer link by the respective ingress PE devices for the traffic flow.

At block 440, the SDN controller selects, from the set of ingress PEdevices based on the respective bandwidth usages of the egress peer linkby the respective ingress PE devices for the traffic flow, a selectedingress PE device.

At block 450, the SDN controller initiates a management action for theselected ingress PE device.

At block 499, method 400 ends.

FIG. 5 depicts a method for use by an ingress provider edge device forredirecting traffic of a traffic flow from a first egress peer link to asecond egress peer link. It is noted that the first and second egresspeer links may be on the same egress peer node or different egress peernodes. It is noted that, although primarily presented in FIG. 5 as beingperformed serially, at least a portion of the functions of method 500 ofFIG. 5 may be performed contemporaneously or in a different order thanas presented in FIG. 5.

At block 501, method 500 begins.

At block 510, the ingress PE device sends, toward a first egress peerlink having a peering relationship with the ingress PE device, a firstdata packet of a traffic flow.

At block 520, the ingress PE device receives, from a controller, acontrol message indicative of a redirection of the traffic flow from thefirst egress peer link to a second egress peer link associated with thefirst egress peer link.

At block 530, the ingress PE device sends, toward the second egress peerlink, a second data packet of the traffic flow.

At block 599, method 500 ends.

It will be appreciated that, although primarily presented herein withrespect to embodiments in which discovery of a set of ingress provideredge devices providing traffic to an egress peer link for a set oftraffic flows is based on a congestion indicator associated with theegress peer link, discovery of a set of ingress provider edge devicesproviding traffic to an egress peer link for a set of traffic flows maybe based on various other types of indicators which may be used as abasis for initiating management actions on the egress peer link (e.g.,Key Performance Indicators (KPIs) or the like).

It will be appreciated that, although primarily presented herein withrespect to embodiments in which discovery of a set of ingress provideredge devices includes discovery of a set of ingress provider edgedevices providing traffic to an egress peer link for a set of trafficflows, in at least some embodiments the discovery of a set of ingressprovider edge devices may include discovery of a set of ingress provideredge devices providing traffic to an egress peer node (e.g., where theegress peer node includes only a single egress peer link such thatdiscovery for the egress peer link may be considered to be discovery forthe egress peer node as a whole, where the egress peer node includesmultiple egress peer links and all of the egress peer links areconsidered as a whole such that the functions may be considered to beprovided for the egress peer node as a whole, or the like).

It will be appreciated that, although primarily presented herein withrespect to embodiments in which discovery of a set of ingress provideredge devices providing traffic to an egress peer link for a set oftraffic flows is based on use of a particular egress peering capability(namely, EPE), discovery of a set of ingress provider edge devicesproviding traffic to an egress peer link for a set of traffic flows maybe based on various other types of peering capabilities and, thus,EPE-specific references herein may be read more generally (e.g., asegress peering or the like).

It will be appreciated that, although primarily presented herein withrespect to embodiments in which discovery of a set of ingress provideredge devices providing traffic to an egress peer link for a set oftraffic flows is based on use of a particular control capability forcontrolling elements of the network (namely, SDN), discovery of a set ofingress provider edge devices providing traffic to an egress peer linkfor a set of traffic flows may be based on various other types ofcontrol capabilities and, thus, SDN-specific references herein (e.g.,SDN controller) may be read more generally (e.g., as control device,control element, or the like).

FIG. 6 depicts a high-level block diagram of a computer suitable for usein performing various functions described herein.

The computer 600 includes a processor 602 (e.g., a central processingunit (CPU), a processor having a set of processor cores, a processorcore of a processor, or the like) and a memory 604 (e.g., a randomaccess memory (RAM), a read only memory (ROM), or the like). Theprocessor 602 and the memory 604 are communicatively connected.

The computer 600 also may include a cooperating element 605. Thecooperating element 605 may be a hardware device. The cooperatingelement 605 may be a process that can be loaded into the memory 604 andexecuted by the processor 602 to implement functions as discussed herein(in which case, for example, the cooperating element 605 (includingassociated data structures) can be stored on a non-transitorycomputer-readable storage medium, such as a storage device or otherstorage element (e.g., a magnetic drive, an optical drive, or thelike)).

The computer 600 also may include one or more input/output devices 606.The input/output devices 606 may include one or more of a user inputdevice (e.g., a keyboard, a keypad, a mouse, a microphone, a camera, orthe like), a user output device (e.g., a display, a speaker, or thelike), one or more network communication devices or elements (e.g., aninput port, an output port, a receiver, a transmitter, a transceiver, orthe like), one or more storage devices (e.g., a tape drive, a floppydrive, a hard disk drive, a compact disk drive, or the like), or thelike, as well as various combinations thereof.

It will be appreciated that computer 600 of FIG. 6 may represent ageneral architecture and functionality suitable for implementingfunctional elements described herein, portions of functional elementsdescribed herein, or the like, as well as various combinations thereof.For example, computer 600 may provide a general architecture andfunctionality that is suitable for implementing one or more of aningress PE device 111, a BR 119, a provider router 120, a providerrouter 130, an SDN controller 140, an ingress PE device discoveryelement 141, or the like.

It will be appreciated that the functions depicted and described hereinmay be implemented in software (e.g., via implementation of software onone or more processors, for executing on a general purpose computer(e.g., via execution by one or more processors) so as to provide aspecial purpose computer, and the like) and/or may be implemented inhardware (e.g., using a general purpose computer, one or moreapplication specific integrated circuits (ASIC), and/or any otherhardware equivalents).

It will be appreciated that at least some of the functions discussedherein as software methods may be implemented within hardware, forexample, as circuitry that cooperates with the processor to performvarious functions. Portions of the functions/elements described hereinmay be implemented as a computer program product wherein computerinstructions, when processed by a computer, adapt the operation of thecomputer such that the methods and/or techniques described herein areinvoked or otherwise provided. Instructions for invoking the variousmethods may be stored in fixed or removable media (e.g., non-transitorycomputer-readable media), transmitted via a data stream in a broadcastor other signal bearing medium, and/or stored within a memory within acomputing device operating according to the instructions.

It will be appreciated that the term “or” as used herein refers to anon-exclusive “or” unless otherwise indicated (e.g., use of “or else” or“or in the alternative”).

It will be appreciated that, although various embodiments whichincorporate the teachings presented herein have been shown and describedin detail herein, those skilled in the art can readily devise many othervaried embodiments that still incorporate these teachings.

What is claimed is:
 1. An apparatus, comprising: a processor and amemory communicatively connected to the processor, the processorconfigured to: identify a traffic flow on an egress peer link of anegress peer node; identify a set of ingress provider edge (PE) devicesthat have peering relationships with the egress peer node; determine,for the ingress PE devices based on respective sets of flow statisticsassociated with the respective ingress PE devices, respective bandwidthusages of the egress peer link by the respective ingress PE devices forthe traffic flow; select, from the set of ingress PE devices based onthe respective bandwidth usages of the egress peer link by therespective ingress PE devices for the traffic flow, a selected ingressPE device; and initiate a management action for the selected ingress PEdevice.
 2. The apparatus of claim 1, wherein, to identify the trafficflow on the egress peer link, the processor is configured to: receive,from the egress peer node, flow statistics associated with the trafficflow on the egress peer link of the egress peer node; and identify thetraffic flow based on the flow statistics associated with the trafficflow on the egress peer link of the egress peer node.
 3. The apparatusof claim 1, wherein, to identify the traffic flow on the egress peerlink, the processor is configured to: receive, from the egress peernode, a plurality of sets of flow statistics associated with arespective plurality of traffic flows supported by the egress peer linkof the egress peer node; and identify the traffic flow, from theplurality of traffic flows supported by the egress peer link of theegress peer node, based on the plurality of sets of flow statisticsassociated with the respective plurality of traffic flows supported bythe egress peer link of the egress peer node.
 4. The apparatus of claim3, wherein the traffic flow comprises one of the plurality of trafficflows having a largest bandwidth on the egress peer link of the egresspeer node.
 5. The apparatus of claim 1, wherein the peering relationshipis based on Egress Peer Engineering (EPE).
 6. The apparatus of claim 1,wherein the processor is configured to: obtain, from the respectiveingress PE devices, the respective sets of flow statistics associatedwith the respective ingress PE devices.
 7. The apparatus of claim 6,wherein, to obtain the respective sets of flow statistics associatedwith the respective ingress PE devices, the processor is configured to:send, toward the respective ingress PE devices, respective flow rulesconfigured to cause the respective ingress PE devices to collect therespective sets of flow statistics for the traffic flow on the egresspeer link of the egress peer node; and receive, from the respectiveingress PE devices, the respective sets of flow statistics for thetraffic flow on the egress peer link of the egress peer node.
 8. Theapparatus of claim 1, wherein, to select the selected ingress PE device,the processor is configured to: select, from the set of ingress PEdevices, one of the ingress PE devices for which the respectivebandwidth usage is largest.
 9. The apparatus of claim 1, wherein, toinitiate the management action, the processor is configured to:initiate, for the selected ingress PE device, redirection of the trafficflow from being directed from the selected ingress PE device to theegress peer link of the egress peer node to being directed from theselected ingress PE device to a second egress peer link of the egresspeer node or a second egress peer node.
 10. The apparatus of claim 9,wherein, to initiate redirection of the traffic flow, the processor isconfigured to: send, toward the selected ingress PE device, a set offlow forwarding rules configured to cause the selected ingress PE deviceto direct traffic of the traffic flow toward the second egress peer linkinstead of toward the egress peer link of the egress peer node.
 11. Theapparatus of claim 1, wherein the apparatus comprises a software definednetworking (SDN) controller.
 12. A method, comprising: identifying, by aprocessor, a traffic flow on an egress peer link of an egress peer node;identifying, by the processor, a set of ingress provider edge (PE)devices that have peering relationships with the egress peer node;determining, by the processor for the ingress PE devices based onrespective sets of flow statistics associated with the respectiveingress PE devices, respective bandwidth usages of the egress peer linkby the respective ingress PE devices for the traffic flow; selecting, bythe processor from the set of ingress PE devices based on the respectivebandwidth usages of the egress peer link by the respective ingress PEdevices for the traffic flow, a selected ingress PE device; andinitiating, by the processor, a management action for the selectedingress PE device.
 13. An apparatus, comprising: a processor and amemory communicatively connected to the processor, the processorconfigured to: identify a set of traffic flows on an egress peer node;identify a set of ingress provider edge (PE) devices that have peeringrelationships with the egress peer node; determine bandwidth usageinformation comprising, for each combination of each of the trafficflows and each of the ingress PE devices, a respective bandwidth usageof the egress peer link by the respective ingress PE device for therespective traffic flow; select, from a set of ingress PE devices basedon the bandwidth usage information, a set of selected ingress PEdevices; and initiate a management action for one of the selectedingress PE devices.